{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 用飛槳+ DJL 實作人臉口罩辨識\n",
    "在這個教學中我們將會展示利用 PaddleHub 下載預訓練好的 PaddlePaddle 模型並針對範例照片做人臉口罩辨識。這個範例總共會分成兩個步驟:\n",
    "\n",
    "- 用臉部檢測模型識別圖片中的人臉(無論是否有戴口罩) \n",
    "- 確認圖片中的臉是否有戴口罩\n",
    "\n",
    "這兩個步驟會包含使用兩個 Paddle 模型，我們會在接下來的內容介紹兩個模型對應需要做的前後處理邏輯\n",
    "\n",
    "## 導入相關環境依賴及子類別\n",
    "在這個例子中的前處理飛槳深度學習引擎需要搭配 DJL 混合模式進行深度學習推理，原因是引擎本身沒有包含 NDArray 操作，因此需要藉用其他引擎的 NDArray 操作能力來完成。這邊我們導入 PyTorch 來做協同的前處理工作:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "// %mavenRepo snapshots https://oss.sonatype.org/content/repositories/snapshots/\n",
    "\n",
    "%maven ai.djl:api:0.12.0\n",
    "%maven ai.djl.paddlepaddle:paddlepaddle-model-zoo:0.12.0\n",
    "%maven ai.djl.paddlepaddle:paddlepaddle-native-auto:2.0.2\n",
    "%maven org.slf4j:slf4j-api:1.7.26\n",
    "%maven org.slf4j:slf4j-simple:1.7.26\n",
    "\n",
    "// second engine to do preprocessing and postprocessing\n",
    "%maven ai.djl.pytorch:pytorch-engine:0.12.0\n",
    "%maven ai.djl.pytorch:pytorch-native-auto:1.8.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ai.djl.Application;\n",
    "import ai.djl.MalformedModelException;\n",
    "import ai.djl.ModelException;\n",
    "import ai.djl.inference.Predictor;\n",
    "import ai.djl.modality.Classifications;\n",
    "import ai.djl.modality.cv.*;\n",
    "import ai.djl.modality.cv.output.*;\n",
    "import ai.djl.modality.cv.transform.*;\n",
    "import ai.djl.modality.cv.translator.ImageClassificationTranslator;\n",
    "import ai.djl.modality.cv.util.NDImageUtils;\n",
    "import ai.djl.ndarray.*;\n",
    "import ai.djl.ndarray.types.Shape;\n",
    "import ai.djl.repository.zoo.*;\n",
    "import ai.djl.translate.*;\n",
    "\n",
    "import java.io.IOException;\n",
    "import java.nio.file.*;\n",
    "import java.util.*;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 臉部偵測模型\n",
    "現在我們可以開始處理第一個模型，在將圖片輸入臉部檢測模型前我們必須先做一些預處理:\n",
    "•\t調整圖片尺寸:  以特定比例縮小圖片\n",
    "•\t用一個數值對縮小後圖片正規化\n",
    "對開發者來說好消息是，DJL 提供了 Translator 介面來幫助開發做這樣的預處理. 一個比較粗略的 Translator 架構如下:\n",
    "\n",
    "![](https://github.com/deepjavalibrary/djl/blob/master/examples/docs/img/workFlow.png?raw=true)\n",
    "\n",
    "在接下來的段落，我們會利用一個 FaceTranslator 子類別實作來完成工作\n",
    "### 預處理\n",
    "在這個階段我們會讀取一張圖片並且對其做一些事先的預處理，讓我們先示範讀取一張圖片:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "String url = \"https://raw.githubusercontent.com/PaddlePaddle/PaddleHub/release/v1.5/demo/mask_detection/python/images/mask.jpg\";\n",
    "Image img = ImageFactory.getInstance().fromUrl(url);\n",
    "img.getWrappedImage();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接著，讓我們試著對圖片做一些預處理的轉換:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NDList processImageInput(NDManager manager, Image input, float shrink) {\n",
    "            NDArray array = input.toNDArray(manager);\n",
    "            Shape shape = array.getShape();\n",
    "            array = NDImageUtils.resize(\n",
    "                            array, (int) (shape.get(1) * shrink), (int) (shape.get(0) * shrink));\n",
    "            array = array.transpose(2, 0, 1).flip(0); // HWC -> CHW BGR -> RGB\n",
    "            NDArray mean = manager.create(new float[] {104f, 117f, 123f}, new Shape(3, 1, 1));\n",
    "            array = array.sub(mean).mul(0.007843f); // normalization\n",
    "            array = array.expandDims(0); // make batch dimension\n",
    "            return new NDList(array);\n",
    "}\n",
    "\n",
    "processImageInput(NDManager.newBaseManager(), img, 0.5f);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上述所見，我們已經把圖片轉成如下尺寸的 NDArray: (披量, 通道(RGB), 高度, 寬度). 這是物件檢測模型輸入的格式\n",
    "### 後處理\n",
    "當我們做後處理時, 模型輸出的格式是 (number_of_boxes, (class_id, probability, xmin, ymin, xmax, ymax)). 我們可以將其存入預先建立好的 DJL 子類別 DetectedObjects 以便做後續操作. 我們假設有一組推論後的輸出是 ((1, 0.99, 0.2, 0.4, 0.5, 0.8)) 並且試著把人像框顯示在圖片上"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "DetectedObjects processImageOutput(NDList list, List<String> className, float threshold) {\n",
    "            NDArray result = list.singletonOrThrow();\n",
    "            float[] probabilities = result.get(\":,1\").toFloatArray();\n",
    "            List<String> names = new ArrayList<>();\n",
    "            List<Double> prob = new ArrayList<>();\n",
    "            List<BoundingBox> boxes = new ArrayList<>();\n",
    "            for (int i = 0; i < probabilities.length; i++) {\n",
    "                if (probabilities[i] >= threshold) {\n",
    "                    float[] array = result.get(i).toFloatArray();\n",
    "                    names.add(className.get((int) array[0]));\n",
    "                    prob.add((double) probabilities[i]);\n",
    "                    boxes.add(\n",
    "                            new Rectangle(\n",
    "                                    array[2], array[3], array[4] - array[2], array[5] - array[3]));\n",
    "                }\n",
    "            }\n",
    "            return new DetectedObjects(names, prob, boxes);\n",
    "}\n",
    "\n",
    "NDArray tempOutput = NDManager.newBaseManager().create(new float[]{1f, 0.99f, 0.1f, 0.1f, 0.2f, 0.2f}, new Shape(1, 6));\n",
    "DetectedObjects testBox = processImageOutput(new NDList(tempOutput), Arrays.asList(\"Not Face\", \"Face\"), 0.7f);\n",
    "Image newImage = img.duplicate(Image.Type.TYPE_INT_ARGB);\n",
    "newImage.drawBoundingBoxes(testBox);\n",
    "newImage.getWrappedImage();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 生成一個翻譯器並執行推理任務\n",
    "透過這個步驟，你會理解 DJL 中的前後處理如何運作，現在讓我們把前數的幾個步驟串在一起並對真實圖片進行操作:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FaceTranslator implements Translator<Image, DetectedObjects> {\n",
    "\n",
    "        private float shrink;\n",
    "        private float threshold;\n",
    "        private List<String> className;\n",
    "\n",
    "        FaceTranslator(float shrink, float threshold) {\n",
    "            this.shrink = shrink;\n",
    "            this.threshold = threshold;\n",
    "            className = Arrays.asList(\"Not Face\", \"Face\");\n",
    "        }\n",
    "\n",
    "        @Override\n",
    "        public DetectedObjects processOutput(TranslatorContext ctx, NDList list) {\n",
    "            return processImageOutput(list, className, threshold);\n",
    "        }\n",
    "\n",
    "        @Override\n",
    "        public NDList processInput(TranslatorContext ctx, Image input) {\n",
    "            return processImageInput(ctx.getNDManager(), input, shrink);\n",
    "        }\n",
    "\n",
    "        @Override\n",
    "        public Batchifier getBatchifier() {\n",
    "            return null;\n",
    "        }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要執行這個人臉檢測推理，我們必須先從 DJL 的 Paddle Model Zoo 讀取模型，在讀取模型之前我們必須指定好 `Crieteria` . `Crieteria` 是用來確認要從哪邊讀取模型而後執行 `Translator` 來進行模型導入. 接著，我們只要利用 `Predictor` 就可以開始進行推論"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Criteria<Image, DetectedObjects> criteria =\n",
    "        Criteria.builder()\n",
    "                .optApplication(Application.CV.OBJECT_DETECTION)\n",
    "                .setTypes(Image.class, DetectedObjects.class)\n",
    "                .optArtifactId(\"face_detection\")\n",
    "                .optTranslator(new FaceTranslator(0.5f, 0.7f))\n",
    "                .optFilter(\"flavor\", \"server\")\n",
    "                .build();\n",
    "   \n",
    "var model = criteria.loadModel();\n",
    "var predictor = model.newPredictor();\n",
    "\n",
    "DetectedObjects inferenceResult = predictor.predict(img);\n",
    "newImage = img.duplicate(Image.Type.TYPE_INT_ARGB);\n",
    "newImage.drawBoundingBoxes(inferenceResult);\n",
    "newImage.getWrappedImage();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如圖片所示，這個推論服務已經可以正確的辨識出圖片中的三張人臉\n",
    "## 口罩分類模型\n",
    "一旦有了圖片的座標，我們就可以將圖片裁剪到適當大小並且將其傳給口罩分類模型做後續的推論\n",
    "### 圖片裁剪\n",
    "圖中方框位置的數值範圍從0到1, 只要將這個數值乘上圖片的長寬我們就可以將方框對應到圖片中的準確位置. 為了使裁剪後的圖片有更好的精確度，我們將圖片裁剪成方形，讓我們示範一下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "int[] extendSquare(\n",
    "        double xmin, double ymin, double width, double height, double percentage) {\n",
    "    double centerx = xmin + width / 2;\n",
    "    double centery = ymin + height / 2;\n",
    "    double maxDist = Math.max(width / 2, height / 2) * (1 + percentage);\n",
    "    return new int[] {\n",
    "        (int) (centerx - maxDist), (int) (centery - maxDist), (int) (2 * maxDist)\n",
    "    };\n",
    "}\n",
    "\n",
    "Image getSubImage(Image img, BoundingBox box) {\n",
    "    Rectangle rect = box.getBounds();\n",
    "    int width = img.getWidth();\n",
    "    int height = img.getHeight();\n",
    "    int[] squareBox =\n",
    "            extendSquare(\n",
    "                    rect.getX() * width,\n",
    "                    rect.getY() * height,\n",
    "                    rect.getWidth() * width,\n",
    "                    rect.getHeight() * height,\n",
    "                    0.18);\n",
    "    return img.getSubimage(squareBox[0], squareBox[1], squareBox[2], squareBox[2]);\n",
    "}\n",
    "\n",
    "List<DetectedObjects.DetectedObject> faces = inferenceResult.items();\n",
    "getSubImage(img, faces.get(2).getBoundingBox()).getWrappedImage();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 事先準備 Translator 並讀取模型\n",
    "在使用臉部檢測模型的時候，我們可以利用 DJL 預先建好的 `ImageClassificationTranslator` 並且加上一些轉換。這個 Translator 提供了一些基礎的圖片翻譯處理並且同時包含一些進階的標準化圖片處理。以這個例子來說, 我們不需要額外建立新的 `Translator` 而使用預先建立的就可以"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var criteria = Criteria.builder()\n",
    "                        .optApplication(Application.CV.IMAGE_CLASSIFICATION)\n",
    "                        .setTypes(Image.class, Classifications.class)\n",
    "                        .optTranslator(\n",
    "                                ImageClassificationTranslator.builder()\n",
    "                                        .addTransform(new Resize(128, 128))\n",
    "                                        .addTransform(new ToTensor()) // HWC -> CHW div(255)\n",
    "                                        .addTransform(\n",
    "                                                new Normalize(\n",
    "                                                        new float[] {0.5f, 0.5f, 0.5f},\n",
    "                                                        new float[] {1.0f, 1.0f, 1.0f}))\n",
    "                                        .addTransform(nd -> nd.flip(0)) // RGB -> GBR\n",
    "                                        .build())\n",
    "                        .optArtifactId(\"mask_classification\")\n",
    "                        .optFilter(\"flavor\", \"server\")\n",
    "                        .build();\n",
    "\n",
    "var classifyModel = criteria.loadModel();\n",
    "var classifier = classifyModel.newPredictor();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 執行推論任務\n",
    "最後，要完成一個口罩識別的任務，我們只需要將上述的步驟合在一起即可。我們先將圖片做裁剪後並對其做上述的推論操作，結束之後再生成一個新的分類子類別 `DetectedObjects`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "List<String> names = new ArrayList<>();\n",
    "List<Double> prob = new ArrayList<>();\n",
    "List<BoundingBox> rect = new ArrayList<>();\n",
    "for (DetectedObjects.DetectedObject face : faces) {\n",
    "    Image subImg = getSubImage(img, face.getBoundingBox());\n",
    "    Classifications classifications = classifier.predict(subImg);\n",
    "    names.add(classifications.best().getClassName());\n",
    "    prob.add(face.getProbability());\n",
    "    rect.add(face.getBoundingBox());\n",
    "}\n",
    "\n",
    "newImage = img.duplicate(Image.Type.TYPE_INT_ARGB);\n",
    "newImage.drawBoundingBoxes(new DetectedObjects(names, prob, rect));\n",
    "newImage.getWrappedImage();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "12.0.2+10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
